feat(ogc): add the water well field operations internal layer - #914
Conversation
Adds water_well_field_operations, an OGC API - Features collection on
/ogcapi-internal only: one feature per water well carrying what a field crew
needs to plan and execute a visit. Every other well layer answers a scientific
question -- water level, chemistry, trend. This one answers an operational one:
may we go to this well, what are we allowed to do when we get there, and is it
overdue?
86 columns: identity and location (including latitude and longitude in decimal
degrees and how the elevation was obtained), construction detail, the current
measuring point, all five status types, the three standing landowner
permissions, monitoring frequency and group membership, manual water-level and
chemistry-sampling statistics, field-visit history, the installed logger and
its continuous record, landowner contacts, and access and directions notes.
Two relations rather than one (migration e1f2a3b4c5d6):
ogc_internal_water_well_field_operations_stats materialized, nightly refresh
Aggregates over observation and transducer_observation.
ogc_internal_water_well_field_operations plain view, served by pygeoapi
Live join of thing, location, status_history, permission_history,
measuring_point_history, monitoring_frequency_history, deployment,
sensor, contact and notes, LEFT JOINed to the stats matview.
The split is not tidiness. Staleness is dangerous on exactly the columns that
are cheap to read: a revoked sampling permission that still reads true until
the next nightly refresh sends a crew onto land they are no longer welcome on.
The current-record rule is also written against CURRENT_DATE, which inside a
materialized view would freeze at refresh time.
Three decisions worth review:
* Internal-only with no public twin. The layer publishes landowner and operator
contact details -- name, organization, role, phone, email -- alongside
staff-written access notes. ogc_water_well_field_operations does not exist
and must never be created. Every credential the internal mount accepts now
reaches personal information, the static desktop-GIS keys included; notices
added to CLAUDE.md and docs/internal-ogc-desktop-gis.md.
* The three permission columns are three-valued. NULL means no permission is on
record, which is not a refusal -- it means nobody has asked the landowner
yet. Collapsing it to false would tell a crew a well is off limits when the
truth is unknown.
* History reads honour end_date, unlike ogc_actively_monitored_wells, which
takes the greatest start_date and lets a status closed in 2019 read as
current. Written up in docs/ogc_conventions.md.
Requested as amp_field_well_operations and renamed before merge per the
pre-merge check in docs/ogc_conventions.md: amp is an unexpanded abbreviation
whose expansion is recorded nowhere in this codebase, AMP labels the layer's
audience rather than filtering its rows (the row set is every water well), and
Group B analytic layers prefer a water_well_ prefix. The AMP crews stay named
in the collection description.
Multi-valued columns are comma-joined text rather than arrays: this layer is
pulled into ArcGIS Pro and QGIS and exported to File Geodatabase and GeoPackage
for offline field use, and neither format has a list type.
Depth to water uses the (value - COALESCE(measuring_point_height, 0))
convention shared with ogc_water_well_summary, ogc_latest_depth_to_water_wells
and ogc_well_water_column, and elevation_method reuses that summary's
provenance lookup verbatim, so the layers cannot disagree.
Twelve supporting indexes come with the migration; none of the foreign keys
these lookups traverse was indexed.
Design rationale in docs/water-well-field-operations-layer.md, column-by-column
sources in docs/water-well-field-operations-columns.md.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two independently-drafted internal water-well layers existed after a side-by-side field comparison: feat/amp-field-operations-ogc-layer and a competing draft, kas-water-well-operations-ogc-layer- bdms-1202. Rather than ship both under colliding migration IDs, this branch (kas-water-well-field-operations-ogc-layer-bdms-1202) started from feat/amp-field-operations-ogc-layer. It already had tests, indexes, docs, and correct date-window current-record semantics. The agreed changes from the comparison follow: - Dropped nma_pk_welldata, county, state, quad_name, elevation_method, nma_formation_zone, and measuring_point_start_date. None of them serve this layer's operational purpose. - Dropped every `_since`/`_reason` column (one per status type, plus monitoring_frequency_since). The current value itself (well_status, monitoring_status, monitoring_frequency, etc.) stays; only the history around it was dropped. - Renamed thing_type, well_casing_materials, well_purposes, measuring_point_height/description, and field_event_last_date to match the naming already established on the public thing-type views (station_type, well_casing_material, well_purpose, mp_height/mp_description, date_last_visited). - Added formation_completion_description and aquifer_system_name, ported from the competing draft. - Restored well_screen to full per-interval detail (screen_depth_top, screen_depth_bottom, screen_description), ported from the competing draft, instead of the count-plus-min/max summary this branch shipped with. A driller doing rehab work needs the actual screened intervals, not an overall range. - Broadened "currently installed equipment" from logger-only to any currently installed sensor, so a camera or barometer is no longer invisible. has_datalogger/datalogger_deployment_count stay logger-scoped for whoever needs that narrower signal. - Published a column per remaining notes.note_type value (13 total), extending the existing access_notes/directions_notes pattern. - Dropped access_status, since status_value has no lexicon terms scoped to Access Status. The column could only ever read NULL, and access_notes already covers the same need. - Wrapped every aggregated equipment and well_screen expression except sensor_type in COALESCE(..., '') before string_agg, so a row with a null value in one field (e.g. a camera with no recording_interval, or a screen with no recorded description) leaves an empty segment instead of silently shortening that column's list and breaking alignment with the others. Config, field descriptions, design docs, and tests updated to match.
Coverage✅ 79.35% total — gate is 75%. Coverage for the Python files changed in this PR
|
tests/features/ogc-cleanup-sprint1.feature's "Internal database relations are separate from public relations" scenario asserts that every ogc_internal_ relation in the schema has a distinct ogc_ public counterpart. That held for every internal-only collection so far (locations, avg_tds_wells, latest_depth_to_water_wells): each is hidden from the public collections catalog, but both relations still exist in the database. water_well_field_operations breaks that pattern on purpose. ogc_water_well_field_operations does not exist at all, by design (see docs/water-well-field-operations-layer.md section 3), since the layer publishes landowner contact details and staff-written notes. The check had no way to know that was intentional, so it failed CI on this PR. Added an explicit exception set for the two relations this layer introduces, rather than loosening the assertion generally. The check should still fail if some future ogc_internal_ relation is missing its public twin by accident.
There was a problem hiding this comment.
🟡 Changes recommended
The new view’s multi-value aggregations and permission_granted_by semantics have verified correctness issues that can misalign positional fields and/or contradict documented meaning.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Adds a new internal-only OGC API Features collection (water_well_field_operations) to serve field crews with an operational “one-stop” well layer (construction + current status/permissions + installed equipment + contacts + notes), backed by a view plus a nightly-refreshed stats materialized view.
Changes:
- Adds Alembic migration creating
ogc_internal_water_well_field_operations(view) andogc_internal_water_well_field_operations_stats(matview) plus supporting indexes. - Publishes the collection on
/ogcapi-internal(pygeoapi internal config) and documents every field incore/ogc-field-descriptions.yml. - Adds comprehensive tests and supporting documentation describing design rationale, column lineage, and internal-only/PII implications.
File summaries
| File | Description |
|---|---|
| tests/test_pygeoapi_mount.py | Marks the new collection as internal-only in mount-related tests. |
| tests/test_ogc.py | Ensures the collection is excluded from the public OGC collections list. |
| tests/test_ogc_water_well_field_operations.py | New test suite validating the layer’s load-bearing semantics and internal-only mounting. |
| tests/test_cli_commands.py | Updates expected matview refresh list/count for CLI refresh command. |
| tests/features/steps/ogc-cleanup-sprint1.py | Exempts the new internal-only-without-public-twin relations from parity checks. |
| services/materialized_views.py | Registers the stats matview for refresh via the CLI path. |
| docs/water-well-field-operations-layer.md | Design doc explaining scope, PII decision, current-record semantics, and wiring. |
| docs/water-well-field-operations-columns.md | Column-by-column source mapping for the new view/matview. |
| docs/ogc_conventions.md | Documents naming and “current-record” semantics conventions for the new layer. |
| docs/internal-ogc-desktop-gis.md | Adds a prominent notice that internal OGC credentials now convey PII due to this layer. |
| core/pygeoapi-config-internal.yml | Publishes the new collection on the internal pygeoapi mount. |
| core/ogc-field-descriptions.yml | Adds per-field titles/descriptions for the new collection. |
| CLAUDE.md | Records the internal-PII nature of /ogcapi-internal and this layer for future changes. |
| alembic/versions/e1f2a3b4c5d6_add_water_well_field_operations_layer.py | Implements the stats matview + feature view and supporting indexes. |
Review details
Suppressed comments (2)
alembic/versions/e1f2a3b4c5d6_add_water_well_field_operations_layer.py:533
- screens has the same positional-alignment requirement as installed_equipment, but orders all three string_agg() columns only by screen_depth_top. If two screens share the same top depth, tie ordering can differ between aggregates and misalign bottom/description segments. Add a stable tie-breaker (e.g., ws.id) to the ORDER BY in all three aggregates.
string_agg(
COALESCE(ws.screen_depth_top::text, ''), '; '
ORDER BY ws.screen_depth_top NULLS LAST
) AS screen_depth_top,
string_agg(
alembic/versions/e1f2a3b4c5d6_add_water_well_field_operations_layer.py:781
- group_names and group_types are intended to be index-aligned, but both string_agg() calls order only by group_name. If two groups share the same name, tie ordering is undefined and the names/types can become misaligned between the two aggregates. Add a deterministic tie-breaker (e.g., group_id).
string_agg(dm.group_name, ', ' ORDER BY dm.group_name)
AS group_names,
string_agg(dm.group_type, ', ' ORDER BY dm.group_name)
AS group_types
- Files reviewed: 14/14 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Summary
Adds
water_well_field_operations, a new internal-only OGC feature layer. This layer contains information a field crew needs to plan and run a visit in one place, instead of having to manually join information from the existing layers.Also includes supporting documentation:
docs/water-well-field-operations-columns.mddocs/water-well-field-operations-layer.mdWhy
This PR addresses the following problem / context:
How
Implementation summary - the following was changed / added / removed:
ogc_internal_water_well_field_operations, a view pygeoapi serves directly, and (2) a nightly-refreshed materialized view that supplies its expensive statistics (manual water-level counts, chemistry-sampling counts, and continuous-reading counts). Status and permission columns stay on the pygeoapi view, so a permission that expires this morning shows as expired immediately.has_dataloggerremains available as a logger-specific flag.Example of the view's fields and values
idnamestation_typerelease_statusalternate_idslatitudelongitudeelevationwell_depthhole_depthwell_casing_diameterwell_casing_depthwell_completion_datewell_driller_namewell_construction_methodwell_pump_typewell_pump_depthformation_completion_codeformation_completion_descriptionwell_purposewell_casing_materialaquifer_system_namescreen_countscreen_depth_topscreen_depth_bottomscreen_descriptionmp_heightmp_descriptionwell_statusmonitoring_statusopen_statusdatalogger_suitability_statusmay_measure_water_levelmay_sample_water_chemistrymay_install_dataloggerpermission_granted_bymonitoring_frequencygroup_namesgroup_typesmanual_water_level_countmanual_water_level_first_datemanual_water_level_last_datedays_since_manual_water_levellast_depth_to_water_ftchemistry_sample_countchemistry_sample_last_datedays_since_chemistry_samplefield_event_countdate_last_visitedhas_dataloggerdatalogger_deployment_countsensor_typemodelserial_nosensor_statusinstalled_daterecording_intervalrecording_interval_unitshanging_point_desccontinuous_reading_countcontinuous_first_datetimecontinuous_last_datetimedays_since_continuous_readingcontact_countprimary_contact_nameprimary_contact_organizationprimary_contact_roleprimary_contact_typeprimary_contact_phoneprimary_contact_emailcontact_namesaccess_notesdirections_notescommunication_notesconstruction_notesmaintenance_noteshistorical_notesgeneral_noteswater_noteswater_quality_notessampling_procedure_notescoordinate_notesowner_comment_notessite_notes_legacypointNotes
Any special considerations, workarounds, or follow-up work to note?